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A METHOD FOR ADAPTIVE DATA/ CONTENT INSERTION IN 
MPEG2 TRANSPORT STREAM 

5 

TECHNICAL FIELD OF THE INVENTION 

The present invention is directed, in general, to data 
delivery systems and, more specifically, to insertion of 
10 localized or personalized content within general data 
during delivery. 

BACKGROUND OF THE INVENTION 

15 Recent advances in digital video technology have made 

possible transmission of locally generated data along with 
video/audio to consumers by insertion into broadcast 
streams. Such capability enables new services and 

applications which enhance the digital television 

20 experience. For example, local broadcast affiliates may 
customize broadcast programs with local advertisements, 
local news and stock information updates (including 
"ticker"' overlays), and local educational programs. 
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Customization may extend down to the level of the 
individual viewer. 

Content insertion/splicing devices exist which 
multiplex data with audio and video to insert local 
5 programs and/or commercials into broadcast streams. 
However, new services enabled by digital video technologies 
require new and efficient ways of adding data streams to 
broadcasts streams within the limit of a given bandwidth. 
An efficient, simple data insertion systems is critical for 

10 timely and opportunistic delivery of such localized and/or 
individualized content to the viewers, one which preferably 
makes the best possible use of available bandwidth. 

For example, the motion pictures expert group (MPEG) 
standard, phase 2 (MPEG-2) is a very flexible standard 

15 permitting easy implementation of new data services without 
changing the underlying standard. Currently the MPEG-2 
transport stream utilizes "null" ■ packets—packets 
identified by the packet type identifier ("PId") number 
8191--as fillers to fill in the bandwidth not utilized by 

20 the audio/video streams or any other useful streams. 
Insertion of data within an MPEG-2 transport stream oy 
replacing null packets (and any other replaceable packets) 
is one option for content insertion. However, simple 
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insertion of data packets for null packets does not allow 
judicious use of available bandwidth, and may not suffice 
to insert data services efficiently without disturbing the 
broadcast stream. 
5 There is, therefore, a need in the art for adaptive 

insertion of content within a digital broadcast streams, 
such as an MPEG-2 transport stream, to enable timely 
delivery of data and maximum utilization of available 
bandwidth without disturbing the broadcast stream in a 
10 destructive manner. 
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SUMMARY OF THE INVENTION 

To address the above-discussed deficiencies of the 
prior art, it is a primary object of the present invention 
5 to provide, for use in receiver, an adaptive data insertion 
mechanism which conducts future available bandwidth 
prediction/estimation by analyzing recent bandwidth in the 
transport stream which is consumed by general programs and 
inserts data by replacement of selected packets within the 

10 transport stream. Estimates of future available bandwidth 
are produced from periodic sampling of bandwidth 
utilization together with information regarding upcoming 
programming changes. The available bandwidth estimate is 
employed to prioritize and schedule data insertion within 

15 the transport stream. Scheduled data is inserted by 
replacement of selected packets within the received 
transport stream before forwarding. 

The foregoing has outlined rather broadly the features 
and technical advantages of the present invention so that 

20 those skilled in the art may better understand the detailed 
description of the invention that follows. Additional 
features and advantages of the invention will be described 
hereinafter that form the subject of the claims of the 
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invention. Those skilled in the art will appreciate that 
they may readily use the conception and the specific 
embodiment disclosed as a basis for modifying or designing 
other structures for carrying out the same purposes of the 
present invention. Those skilled in the art will also 
realize that such equivalent constructions do not depart 
from the spirit and scope of the invention in its broadest 
form. 

Before undertaking the DETAILED DESCRIPTION OF THE 
INVENTION below, it may be advantageous to set forth 
definitions of certain words or phrases used throughout 
this patent document: the terms "include" and "comprise/' 
as well as derivatives thereof, mean inclusion without 
limitation; the term "or" is inclusive, meaning and/or; the 
phrases "associated with" and "associated therewith," as 
well as derivatives thereof, may mean to include, be 
included within, interconnect with, contain, be contained 
within, connect to or with, couple to or with, be 
communicable with, cooperate with, interleave, juxtapose, 
be proximate to, be bound to or with, have, have a property 
of, or the Like; and the term "controller" means any 
device, system or part thereof that controls at least one 
operation, whether such a device is implemented in 
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hardware, firmware, software or some combination of at 
least two of the same. It should be noted that the 
functionality associated with any particular controller may 
be centralized or distributed, whether locally or remotely. 
Definitions for certain words and phrases are provided 
throughout this patent document, and those of ordinary 
skill in the art will understand that such definitions 
apply in many, if not most, instances to prior as well as 
future uses of such defined words and phrases. 
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BRIEF DESCRIPTION OF THE DRAWINGS 

For a more complete understanding of the present 
invention, and the advantages thereof, reference is now 
5 made to the following descriptions taken in conjunction 
with the accompanying drawings, wherein like numbers 
designate like objects, and in which: 

FIGURE 1 depicts a data transmission system in which 
adaptive content insertion within a broadcast stream is 
10 implemented according to one embodiment of the present 
invention ; 

FIGURE 2 illustrates in greater detail a system for 
adaptive content insertion in a broadcast stream according 
to one embodiment of the present invention; 
15 FIGURE 3 depicts a high level flow chart for a process 

of adaptive content insertion in a broadcast stream 
according to one embodiment of the present • invention; 

FIGURE 4 depicts a high level flowchart for a process 
of available bandwidth prediction or estimation during 
20 adaptive content insertion within a broadcast stream 
according to one embodiment of the present invention; 

FIGURE 5 depicts a high level flowchart for a process 
of prioritization and data scheduling during adaptive 
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content insertion in a broadcast stream according to one 
embodiment of the present invention; and 

FIGURE 6 depicts a high level flowchart for a process 
of data insertion during adaptive content insertion in a 
broadcast stream according to one embodiment of the present 
invention . 
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DETAILED DESCRIPTION OF THE INVENTION 

FIGURES 1 through 6, discussed below, and the various 
embodiments used to describe the principles of the present 
invention in this patent document are by way of 
illustration only and should not be construed in any way to 
limit the scope of the invention. Those skilled in the art 
will understand that the principles of the present 
invention may be implemented in any suitably arranged 
device . 

FIGURE 1 depicts a data transmission system in which 
adaptive content insertion within a broadcast stream is 
implemented according to one embodiment of the present 
invention. Data transmission system 100 includes a 

transmitter 101 and a receiver 102 coupled by a 
communications link 103. Communications link 103 may be a 
wireless, air interface communications channel or a wired, - 
packet- or circuit-switched communications network channel 
in accordance with the known art. In the exemplary embodi- 
ment, transmitter 101 and receiver 102 are a high 
definition television (KDTV) transmitter and receiver, 
respectively. Alternatively, however, transmitter 101 and 
receiver 102 may be any transmitter/receiver employed 
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within a data transmission system (including voice, data or 
voice and data communications systems) in which local 
content insertion is performed. 

Transmitter 101 in the exemplary embodiment includes a 
5 remote transmitter 110 and a local transmitter 111 coupled 
by a communications link 112. The terms "local" and 
"remote" are not intended to imply geographic location, but 
instead describe the relationship of the respective 
transmitters 111 and 110 in the transport chain with regard 

10 to the receiver 102--that is, local transmitter 111 is 
closer, within the transport chain, to receiver 102 than is 
remote transmitter 110. As with communications link 103, 
communications link 112 may be a wireless, air interface 
communications channel or a wired, packet- or circuit- 

15 switched communications network channel in accordance with 
the known art. 

FIGURE 2 illustrates in greater detail a system for 
adaptive content insertion in a broadcast stream according 
to one embodiment of the present invention. Local 
20 transmitter (or, more accurately, re-transmitter) 111 
receives the initial transport stream 200 from remote 
transmitter 110 and inserts content to form a new transport 
stream 201, which is then transmitted to the receiver. 
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A good data insertion system should satisfy the 
following requirements: (1) data insertion without 

destructive disturbance of current programs; (2) timely 
data delivery; and ( 3 ) maximization of available bandwidth 
5 utilization. The present invention comprises an adaptive 
data insertior system for embedding multimedia application 
dataand/or any other text data in an MPEG-2 transport 
stream and consists of two phases: bandwidth predict- 
ion/estimation based upon analysis of recent bandwidth 
10 consumed by programs of the transport stream, and insertion 
of data by replacement of selected packets (e.g., null 
packets or audio layer packets) representing the measure of 
available bandwidth . 

A bandwidth (BW) estimator 202 within local 
15 transmitter 111 tracks the bandwidth consumed by each 
packet type identifier (PId) , including replaceable packets 
such as null packets, counting the number of packets with 
each different packet type identifier during a given 
' period. These statistics are then utilized to predict the 
20 future available bandwidth within the transport stream. 

The estimate of future available bandwidth is received 
and employed by data scheduler 203 to prioritize and 
schedule the data streams which are to be inserted into the 
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incoming initial transport stream 200. For this purpose, 
data scheduler 203 receives the data description 204, a 
profile of the data to be inserted within the incoming 
initial transport stream 200 which includes, for example, 
5 the name and or uniform resource locator (URL) for the data 
file to be inserted, the required (or desired) bit rate for 
transmission, the desired schedule for insertion, etc. The 
data to be inserted may be multimedia content such as a 
joint photographic experts group (JPEG) standard graphical 

10 image or an audio sound track file, or any other binary 
(byte-code) or text data. 

Data scheduler 203 prioritizes and schedules data 
streams to be inserted into the incoming transport stream 
200 based upon the predicted available bandwidth obtained 

15 from bandwidth estimator 202 and the required/desired bit 
rate obtained from the data description 204 so that the bit 
rate required by a particular data stream or resource is 
satisfied . 

Once the data streams are scheduled, a packet inserter 
20 205 within the local transmitter 111 watches for 
replaceable packets (e.g., null packets) within the 
incoming transport stream 200 and replaces those packers 
with data packets of the currently scheduled insertion data 
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within new transport stream 201. Packet inserter 205 
contains a list of packet types, identified by packet type 
identifier, which may be replaced within the incoming 
transport stream 200. The system may be configured with 
5 such a list by a local administrator at the local 
transmitter. Non-replaceable packets within the incoming 
transport stream 200 are simply passed through packet 
inserter 205 into the new transport stream 201. 

The insertion data streams (data streams comprising 

10 the content to be inserted within the incoming transport 
stream 200) must be packaged within a data format 
consistent with that employed by the incoming transport 
stream 200, which in the exemplary embodiment is an MPEG-2 
data format. Within a high-level MPEG2 data format, there 

15 may be multiple levels of data encapuslation, depending on 
the application. Data generator 206 obtains the raw data 
207 for the content to be inserted and * encapsulates the 
data in an MPEG-2 data format, such as an MPEG-2 section or 
packet (i.e., 188 byte packets). Data generator 206 need 

20 not necessarily form part of local transmitter 111, but 
instead could be an external entity preparing the data 
packets in a specified format. Similarly, data description 
204 and data files 207 for the insertion content need not 
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originate locally within local transmitter 111, but may 
instead be retrieved from a remote facility utilizing, for 
example, and Internet connection. However, data generator 
206, data description 204, and data files 207 may 
5 optionally form part of and/or be stored within local 
transmitter 111. 

FIGURE 3 depicts a high level flow chart for a process 
of adaptive content insertion in a broadcast stream 
according to one embodiment of the present invention. The 

10 process 300, performed by local transmitter 111 in FIGURE 
2, begins with a request being received (step 301) by a 
local transmitter for future content insertion in a 
transport stream being relayed by the local transmitter. 
An estimate or prediction of future bandwidth (step 302) 

15 available for future content insertion is first conducted 
based on current characteristics of the transport stream. 
The data for the requested content insertion is then 
prioritized and scheduled (step 303) based on the 
predicted/estimated future available bandwidth. The data 

20 for the requested content insertion may optionally be 
encapsulated (step 304) concurrently with the steps of 
bandwidth estimation and prioritization/scheduling, either 
within the local transmitter or within a remote entity. 



US010134APP.doc 



14 



PATENT 

Alternatively, encapsulation of the insertion data may be 
performed before or after the steps of bandwidth estimation 
and prioritization/scheduling. 

The data for the insertion content is then inserted 
within the transport stream as possible (step 305) by 
replacement of replaceable packets within the incoming 
transport stream, and the modified transport stream is 
forwarded to the receiver or next re-transmitter. The 
process then becomes idle (step 306) until a subsequent 
content insertion request is received. Those skilled in 
the art will recognize that multiple content insertion 
requests may be processed as described above in a 
concurrent or overlapping (either fully or partially) 
manner . 

FIGURE 4 depicts a high level flowchart for a process 
of available bandwidth prediction or estimation during 
adaptive content insertion in a broadcast stream according, 
to one embodiment of the present invention. The process 
400, corresponding to step 302 in FIGURE 3 and performed by 
bandwidth estimator 202 in FIGURE 2, begins with bandwidth 
estimation being initiated (step 401) . Initially, 
bandwidth utilization is measured (step 4 02) over a short 
period of time such as, for example, 1 second 
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(approximately 12,830 MPEG-2 transport stream packets for a 
High Definition Digital Television channel) . Bandwidth 
utilization is measured by counting the number of packets 
for each unique packet type identifier which are detected 
5 during the measurement interval , then computing the number 
of replaceable packets within the transport stream during 
the measured interval. 

For television broadcast systems, which are the 
subject of the exemplary embodiment, bandwidth utilization 

10 is more or less uniform throughout a particular program, 
but often undergoes a sharp change when switching from one 
program to another (e.g., from a news program to an action 
movie or vice versa) . Such changes may be predicted, 
however, by analyzing system information tables and/or 

15 electronic program guide (EPG) data (step 403) such as the 
event information table (EIT) which provides the lineup of 
current and upcoming programs. Resources similar to the 
system information tables, such as history tables tracking 
bandwidth utilization as a function of the time of day, may 

20 be employed for other environments. 

Using the analysis of the system information tables 
together with the calculation of replaceable packets, an 
estimate or prediction of the future available bandwidth is 
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produced (step 404). Since bandwidth measurement is 

conducted periodically, the interval between sampling 
periods is allowed to elapse (step 405) before repeating 
the bandwidth utilization measurement step and generating 
an updated estimate of future available bandwidth. The 
process may continue until terminated by an external 
process . 

Experiments conducted on Advanced Television Systems 
Committee (ATSC) high definition (HD) content reveal that 
the average bandwidth utilization of a television program 
is, to some degree, related to the type of program. For 
example, the average bandwidth utilization of a news 
program is different from that of a movie. Combined with 
the programming information within the EIT, the recent 
(last one-minute sample) statistics may be employed to 
estimate the bandwidth available for the next program, 
which usually lasts about 30 minutes. Accordingly, if 
estimates are generated in ten minute intervals, the 
estimate may, depending upon the time of the estimate, be 
useful for the next 10, 20 or 30 minutes. 

The absolute value of the bandwidth availability 
estimate depends on the video quality of the program. 
Relative bancwidth information between two programs is 



US010134APP.doc 



17 



PATENT 

employed to predict the future bandwidth, with some 
exceptions such as commercials embedded within a television 
program. Accuracy of the bandwidth availability estimates 
may be improved by discarding spurious peaks and valleys 
within the (ongoing) bandwidth estimation process, and/or 
by combining recognition of such spurious peaks with other 
commercial detection (cut-detection) methods to derive more 
accurate information regarding the overall bandwidth 
utilization of a particular program. 

FIGURE 5 depicts a high level flowchart for a process 
of prioritization and data scheduling during adaptive 
content insertion in a broadcast stream according to one 
embodiment of the present invention. The process 500, 
corresponding to step 303 in FIGURE 3 and performed by data 
scheduler 203 in FIGURE 2, begins with receipt of a 
bandwidth estimate (step 501) from the bandwidth estimator. 
The data description for the requested insertion content is 
then analyzed (step 502) and a determination is made (step 
503) as to whether sufficient bandwidth is available for 
insertion of the specified content (or contents) given the 
associated bandwidth, timing and bit rate requirements. 

The data description will typically consists of: the 
name (and UPL) of the data file for the content to be 
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inserted (the scheduler may pre-fetch content if the data 
is not local) ; the size of the data file; the desired (or 
required) bit rate; the frequency of insertion, if the 
content is to be transmitted within the transport stream 
5 repeatedly; the order of sending data streams for specific 
insertion content (e.g., sending data stream 1 before 
sending data stream 2 at the same bit rate) ; and the 
urgency or priority of the insertion content. From the 
above parameters, a determination may be made if sufficient 

10 bandwidth is estimated to be available for insertion. 

If the bandwidth availability estimate predicts that 
sufficient bandwidth will be available during the relevant 
period for insertion of at least one block of insertion 
content based on the size of the respective insertion 

15 content (bandwidth requirement) and the desired bit rate, 
insertion of the respective content is scheduled (step 
504). Where insertion of multiple blocks of content is 
requested, each block is scheduled for insertion based on 
the associated urgency/priority from the data description 

20 for that block, the (remaining) available bandwidth within 
rhe bandwidth availability estimate, and the particular bit 
rate requirements for the respective block of content 
(obtained from the data description) . 
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If the predicted available bandwidth is insufficient 
for insertion of a requested block of content without 
degradation of or destruction within the content quality 
for the incoming transport stream, insertion is not 
scheduled for that content. However, an optional override 
may be implemented for urgent local information such as 
updating local severe weather information, preferably with 
minimal impact on the quality for the broadcast stream into 
which the content is inserted. Similarly, an override for 
scheduled content may be implemented for higher priority 
insertion content for which insertion is requested after 
scheduling of previously requested insertion content. 

The data scheduler may also be required to prepare 
data transport stream packets (188 byte packets) for 
content scheduled to be inserted (step 505) and then 
becomes idle (step 506) until an updated future bandwidth 
availability estimate (or new request for content 
insertion) is received. 

FIGURE 6 depicts a high level flowchart for a process 
of data insertion during adaptive content insertion in a 
broadcast stream according to one embodiment of the present 
invention. The process 600, corresponding to step 305 in 
FIGURE 3 and performed by packet inserter 205 in FIGURE 2, 
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begins with content being scheduled for insertion (step 

601) . Incoming transport stream packets are checked (step 

602) to determine if the packet is a replaceable packet 
(step 603), as identified by a packet type identifier which 
has been designated as corresponding to a replaceable 
packet (such as a null packet) . 

If a particular packet is found to be a replaceable 
packet, a packet from the packet stream scheduled and 
prepared by the data scheduler is inserted (step 604) in 
place of the replaceable packet within the transport stream 
forwarded to the receiver. Additionally, the relevant 
system information—mostly within system information 
packets such as the program map table (PMT) packets— should 
be modified to include identification of the newly inserted 
'.data stream(s). Modification of system packets is 

performed by either sending a new systems information table 
with an updated version number or by replacing the existing 
systems tables with utilizing new systems tables packets. 

Care must be taken during packet insertion to ensure 
that packet type identifiers assigned to new data streams 
do not conflict with existing packet type identifiers. 
Some packet type identifier information for the inserted 
data stream(s) may not appear in the program map table, but 
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may appear in other tables instead. For instance, some 
data streams specific to an application may be sent without 
including the packet type identifier within standard tables 
such as the program map table, such as when a service 
provider sends a data stream with a "hard-coded" packet 
type identifier understood only by some receivers, a common 
case given the flexibility of MPEG-2 the systems layer. 

The present invention provides a general purpose 
system for content insertion and replacements in an MPEG-2 
transport stream. This may be implemented in C + + using 
object-oriented methodology to be very flexible and usable 
in various configurations such as the remultipiexer , 
demultiplexer, etc., and is easily extendable to 
accommodate new protocol stacks. 

Although disclosed in the context of MPEG-2 systems, 
the present invention is also suitable for inserting data 
into MPEG-4 streams, particularly since MPEG-4 is based on 
scalable, object level encoding requiring finer granularity 
of data insertion. The present invention is also relevant 
to MPEG-7 based applications in the context of meia data 
insertion . 

The present invention is motivated by the need for 
inserting low-bandwidth data services in a timely manner 
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without disturbing the broadcast stream while judiciously 
using available bandwidth. For example, data services 
including HyperText Markup Language (HTML) pages, 
Extensible Markup Language (XML) files, and downloadable 
5 files associated with video content require estimation of 
available bandwidth for prioritization of data services and 
timely delivery because some data services are tightly 
synchronized with video content. 

Numerous applications and systems may advantageously 

10 employ the present invention. Insertion of application 
data such as Java class files, image files, triggers or 
text data associated with an application/user service may- 
employ the present invention. Similarly the present 
invention may be utilized to insert local advertisement 

15 information into a transport stream, or to replace a 
general advertisement stream with a localized or 
personalized advertisement stream. Replacement of an audio 
stream with the audio content in a different language, 
although requiring tighter synchronization between the 

20 audio and video, may be performed utilizing the present 
invention, assuming that the insertion audio stream has 
approximately the same timing as the stream being replaced. 
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Closed-caption information in a different language may also 
be inserted . 

It is important to note that while the present 
invention has been described in the context of a fully 
functional hard-ware based system and/or network, those 
skilled in the art will appreciate that the mechanism of 
the present invention is capable of being distributed in 
the form of a machine usable medium containing instructions 
in a variety of forms, and that the present invention 
applied equally regardless of the particular type of signal 
bearing mediam utilized to actually carry out the 
distribution. Examples of machine usable mediums include: 
nonvolatile, hard-coded type mediums such as read only 
memories (ROMs) or erasable, electrically programmable read 
only memories (EEPROMs) , recordable type mediums such as 
floppy disks, hard disk drives and compact disc read only 
memories (CD-ROMs) or digital versatile discs (DVDs), and 
transmission type mediums such as digital and analog 
communication links. 

Although the present invention has been described in 
detail, those skilled in the art will understand that 
various changes, substitutions and alterations herein may 
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be made without departing from the spirit and scope of the 
invention in its broadest form. 
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